Reliability of Data Collection Methods in Social Media Research

In this paper we argue for the systematic assessment and reporting of the reliability of data collection methods that rely on the automated collection of data from third party social media sites, where the researcher does not have direct access to meaning or control over data quality. Building on a long tradition of reliability assessment in the sciences and specifically in social science, we propose methods for the assessment of reliability, also for textual data that is increasingly mined from social media for the purposes of studying online populations.